Eliciting Requirements for a Adaptive Decision Support System through Structured User Interviews

Eliciting user requirements at an early stage of software development can safe development time and effort. However, identify requirements for adaptivity, such as inter-individual differences in needs or preferences is not trivial. In this paper we revisit results reported in a previous paper from a methodological point of view. Using an example, we argue that scenarios in combination with structured interviews are not able to adequately identify adaptivity requirements due to reasons inherent to the method, such as the users’ trust and their ability to anticipate system funtionality. We suggest that more implicit methods must be used at early development phases to obtain unbiased results. 1 User-Centered Design and Adaptivity User-adaptive systems are by definition interactive systems. While there is only little literature on guiding the development of user-adaptive systems from a software engineering point of view (e.g., [8]), user-centered design techniques seem to be an obvious process to be adopted for the design of this kind of systems. The rational of user-centered design (UCD) is to place the person as opposed to the software artifact at the center [7]. Users are involved in the development process in very early phases of the software development and in fact throughout the complete development lifecycle. Involving users from the very beginning can help to discover their mental models and expectations, to identify and analyze theory tasks, workflow and goals, and in general to validate the developers’ assumptions about the users. As UCD focuses on cognitive factors (such as perception, memory, learning, problem-solving, etc.) it seems particularly suitable for user-adaptive systems. In this paper we explore the suitability of one particular technique for eliciting user requirements: scenarios. We argue that scenarios used in combination with structured interviews are not able to adequately identify adaptivity requirements due to reasons inherent to the method. Using the example of the development of a decision support system, we show that scenarios did not inform the development process in the expected way. Reasons for this shortcoming are discussed. 2 Informing the Development of a Decision-Support System In a previous paper [6] we outlined the development of a Decision-Support System in the domain of software engineering. 2.1 Software engineering decision support Software engineering decision support (SE-DS) is an emerging field [9,10]. One of the major goals of SE-DS is to support software managers in selecting suitable SE technologies. Suitability implies the existence of a defined level of evidence about the effectiveness of a specific SE technology in a given context. In summary, a SE-DS aims at providing managers in the software process with a comprehensive overview of the state-of-the-art and the state-of-the-practice in software engineering in order to facilitate decisions upon new methods and techniques to be introduced. Software engineering roles that might benefit from decision support include the project manager, the product manager, and the quality manager. 2.2 Adaptivity Hypothesis We set up a pilot study in order to elicit requirements for such a system. We expected that the systems’ presentation of the results should be tailored to these different roles. For example, while project managers might be more interested in the impact of a particular method on project results in general, quality managers might have a focus on the potential to detect or reduce error rates. We anticipated a system that might learn from interaction what items to recommend to users in different roles using a case-based reasoning (CBR) approach [1]. Similar user modeling techniques have been shown to be effective in adaptive sales support [4,12] as well as adaptive decision support for IT security tasks [5]. 2.3 Data Collection Method A number of data collection techniques have been suggested to inform the requirements phase when developing a user-adaptive system [2], including interviews, questionnaires, focus groups, systematic observations, task analysis, cognitive and socio-technical models, contextual inquiry, participative evaluation, ethnography. As we had no existing system or early prototype available that could be analyzed in focus groups or be the basis of a task analysis or contextual inquiry, we decided to assess the preferences and goals of software managers with structured interviews. In order to help interviewees imagine concrete decision support tasks and situations in which a comprehensive SE-DSS might (or might not) be helpful, we offered three scenarios. A scenario consisted of a common part to set the scene of management decision-making (i.e., what kind of information can be obtained, what is the basis for decision support, what is not avail-able), and specific parts linked to one of three particular roles: (1) quality manager, (2) project manager, and (3) product manager. By introducing these scenarios we aimed at finding differences in the user requirements of the three roles. If that was the case, the development of a comprehensive SE-DSS had to take these differences into consideration by modeling these differences. The interview questions were developed in collaboration with an expert in cognitive psychology. Question 1 aimed at eliciting reasons for using a comprehensive SE-DSS (motivation). Question 2 aimed at identifying benefits of a comprehensive SE-DSS for improvement management on the organizational level. Question 3 aimed at getting an idea of the amount of user interaction that could be expected. Question 4 aimed at identifying the types of information that users need for comprehensive SE-DS. Question 5 aimed at prioritizing the different types of information needed by the users. Question 6 aimed at getting a better understanding about how query results should be presented to the user. Question 7 aimed at identifying other application areas (not mentioned in the scenarios) of comprehensive SE-DSS. The questions were not only aimed at eliciting requirements from potential future users of a comprehensive SE-DSS, but also to answer the question if a user modeling approach should be used for implementation. Additionally, the questions were used to substantiate the validity of the scenarios offered to the interviewees. Obviously, from a user modeling point of view, questions 3, 4 and 5 are the most relevant for creating a user model while we did not expect differences across roles for the other questions.